Add verbose Learning Mode denial logging by richiemsft · Pull Request #916 · microsoft/mxc

Richie Gomez (richiemsft) · 2026-08-14T21:42:38Z

📖 Description

Adds deterministic verbose logging for every successful Learning Mode denial analysis while preserving the canonical policy-output contract and expanding canonical decoding for newly validated ETL shapes.

Each analysis now produces a transactional pair:

denials.json retains canonical, actionable denials, including complete file paths.
denials.verbose.json retains a bounded, deduplicated diagnostic superset. Complete file paths are replaced with <REDACTED>; standalone account names and username components in non-file identifiers are also redacted before bounding or hashing.

The verbose logging output includes canonical denial occurrences and closed classifications for outcomes omitted from canonical policy generation, including unusable resource paths, unsupported object types and event schemas, malformed payloads, decoder limits, and unresolved capability events. Stable non-file identifiers such as SIDs, capability names, provider GUIDs, PIDs, and named-object details remain available for diagnosis.

The guarded WPR path now preserves known-provider events through process-scoped selection and replay. Brokered event 28 is scoped using its payload ProcessId, including partial-decode recovery, rather than the broker's ETW header PID.

Native capture and plm stop share a transactional paired-output writer. Both JSON files are staged and synchronized before no-clobber promotion, with ownership- and digest-aware rollback. Audit mode consumes these capture artifacts directly and relocates both the JSON pair and retained ETL through no-clobber staging with identity- and digest-verified source cleanup, keeping metadata truthful without deleting files owned by another process.

Additional ETL findings

Validation across eight new namespace scenarios (16 retained ETLs: block and allow) identified additional canonical decoder shapes:

Event 28 is schema-dependent. UI-shaped payloads containing Category and Detail now emit canonical UI denials such as resource: "Handles", resourceType: "ui", rather than incorrectly treating the payload's package SID as a capability.
Named Section, SymbolicLink, and Timer access checks now emit canonical other resources with object-specific access-mask classification.
The UI Handles scenario represents JOB_OBJECT_UILIMIT_HANDLES, primarily cross-process HWND/USER-object handle access.
SCM, ETW provider/session, and power-policy scenario-specific checks still expose empty ObjectType and ObjectName. They remain bounded verbose logging diagnostics because the ETL provides no stable canonical resource identifier.

The updated decoder increased canonical coverage in 15 of the 16 supplied ETLs. The remaining ETW Session block trace was an early process-initialization failure and contained none of the newly supported named shapes.

🔗 References

Builds on the Learning Mode capture hardening from #847.
This is for issue #828

🔍 Validation

learning_mode_core: 43 tests passed
learning_mode_windows: 224 tests passed
appcontainer_common: 247 tests passed
plm: 226 tests passed, 1 ignored
wxc: 40 tests passed
Clippy passed with warnings denied for learning_mode_core, learning_mode_windows, and plm
Release wxc-exec.exe and plm.exe build
Guarded-WPR VM validation with canonical and typed verbose logging groups, zero overflow, and no username leakage
Production lm_analyze decoder run against all 16 supplied retained ETLs
Confirmed canonical ui/Handles, Timer, Section, and SymbolicLink output; confirmed event 28 UI no longer emits a bogus package-SID capability
Fixture accounting now follows the production handler exactly; out-of-scope events do not consume the processing budget
Create-new output pairs no longer create persistent per-run lock files; relocation publishes committed destinations and surfaces source-cleanup warnings
Malformed event 28 diagnostics classify UI/capability resources only when the schema name identifies the payload shape
GitHub Actions build, lint, integration, CodeQL, CLA, and dependency-feed checks passed before the latest ETL decoder commit; replacement checks run automatically

✅ Checklist

Signed the Contributor License Agreement
Linked to an issue
Updated documentation (if applicable)
Updated Copilot instructions (if build, architecture, or conventions changed)
If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

Bug fix
Feature
Task

GitHub Actions runs the PR validation build automatically. The ADO pipeline
(MXC-PR-Build) is the Azure version of the PR pipeline, kept in parity with the GitHub
Actions build; it runs on merge to main, and Microsoft reviewers with write access can trigger it
on a PR with /azp run. See docs/pull-requests.md.

If the dependency-feed-check check fails on a new dependency, the crate must be added to
the feed before the PR can pass. See docs/pull-requests.md
for the steps.

Microsoft Reviewers: Open in CodeFlow

azure-pipelines · 2026-08-14T21:42:48Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Route wxc-exec --audit through captureDenials so native PSEC/V2 remains preferred and guarded WPR provides compatible analysis and ETL retention. Reuse canonical denials for adjusted policy generation and support retainEtl consistently across both capture providers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Update repository guidance for captureDenials-backed audit routing and guarded retainEtl parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Emit a bounded, username-redacted Data Loop JSON sibling containing canonical denial occurrences and classified decoder outcomes. Keep paired output transactional across native, guarded, and audit capture paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Copilot

Pull request overview

Adds bounded, redacted Data Loop diagnostics alongside Learning Mode denial output and integrates them across native capture, guarded WPR, and audit workflows.

Changes:

Adds the Data Loop model, aggregation, sanitization, and paired output.
Adds process-scoped guarded-WPR ETL relogging and retention.
Reworks --audit to consume captureDenials artifacts directly.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`.github/copilot-instructions.md`	Updates Learning Mode architecture guidance.
`README.md`	Revises audit-mode documentation.
`docs/learning-mode/capabilities.md`	Documents Data Loop and retained ETL behavior.
`src/Cargo.lock`	Records the new dependency edge.
`src/backends/appcontainer/common/src/appcontainer_runner.rs`	Supports guarded ETL transfer.
`src/backends/appcontainer/common/src/base_container_runner.rs`	Integrates retained guarded captures.
`src/backends/appcontainer/common/src/capture_output.rs`	Adds paired denial/Data Loop output.
`src/backends/appcontainer/common/src/guarded_capture.rs`	Extends guarded-capture interfaces.
`src/backends/learning_mode/windows/src/capability_dacl.rs`	Adds diagnostic metadata to recovered denials.
`src/backends/learning_mode/windows/src/etl_decode.rs`	Aggregates Data Loop outcomes and selects relog events.
`src/backends/learning_mode/windows/src/etl_filter.rs`	Adds process-scoped ETL relogging.
`src/backends/learning_mode/windows/src/extractors.rs`	Adds sanitization and diagnostic classification.
`src/backends/learning_mode/windows/src/lib.rs`	Exports ETL filtering.
`src/backends/learning_mode/windows/src/tdh_decode.rs`	Categorizes recoverable decoding failures.
`src/core/learning_mode_core/src/analyze.rs`	Carries and bounds Data Loop results.
`src/core/learning_mode_core/src/data_loop.rs`	Defines the Data Loop model and writer.
`src/core/learning_mode_core/src/lib.rs`	Exports Data Loop APIs.
`src/core/learning_mode_core/src/model.rs`	Makes classifications sortable.
`src/core/mxc_engine/src/guarded_capture.rs`	Implements guarded trace transfer.
`src/core/mxc_engine/src/lib.rs`	Exports audit runner resolution.
`src/core/mxc_engine/src/run.rs`	Adds audit-specific resolver entry point.
`src/core/wxc/Cargo.toml`	Adds Learning Mode core dependency.
`src/core/wxc/src/audit.rs`	Relocates and post-processes capture artifacts.
`src/core/wxc/src/main.rs`	Routes audit through `captureDenials`.
`src/host/plm/readme.md`	Documents guarded analysis and validation.
`src/host/plm/src/analysis.rs`	Returns canonical documents after writing.
`src/host/plm/src/elevated.rs`	Adds filtered analysis/trace transfer protocol.
`src/host/plm/src/log.rs`	Consumes guardian-returned analysis directly.
`src/host/plm/src/secure_scratch.rs`	Manages filtered ETL scratch storage.
`src/host/plm/src/stop.rs`	Extracts canonical-denial post-processing.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Keep excluded namespace evidence uniquely identifiable, retain guarded-trace unknown schemas and brokered capability events, and emit canonical/Data Loop outputs as a transactional pair. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Use targeted broker PID decoding without aborting on unrelated malformed payloads, and make rollback ownership-aware, portable, and no-clobber. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Preserve malformed scoped capability events, retain uncached TDH schemas after the cache cap, and make rollback recovery no-clobber and deletion tolerant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Detect in-place overwrites before deleting a promoted output and directly exercise TDH short-circuit and cache-capacity behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Restore backups even when quarantined output cleanup fails, preserve both errors, and hash promoted files from a known offset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Recover the brokered payload ProcessId when full event decoding fails after that property, and drop only events that remain unscopable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Surface quarantine restoration failures with accurate rollback context and codify malformed capability PID fallback scoping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Identify backup, quarantine, and destination paths in cleanup and restoration failures while preserving strict error propagation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acb632fc-7b75-4d0e-9be5-f11de51652bf

Copilot

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/core/learning_mode_core/src/data_loop.rs:184

Once the signature set contains only canonical groups, every new distinct canonical event calls evict_one_noncanonical_group, which scans all 4,096 entries before overflowing it. At the 1,000,000-event bound this can perform billions of comparisons, turning an intentionally bounded diagnostic path into a CPU bottleneck. Track whether any noncanonical group exists (or partition/index the two classes) so the full scan is skipped when eviction cannot succeed.

        while self.signatures.len() >= MAX_DATA_LOOP_GROUPS
            || retained_bytes.saturating_add(serialized_len) > max_bytes
        {
            if !signature.reason.is_canonical_denial()
                || !self.evict_one_noncanonical_group(Some(retained_bytes))
            {

src/host/plm/src/elevated.rs:2341

This helper is now also used for guarded captureDenials retention, so create_dir_all makes a missing outputPath parent succeed only when guarded WPR plus retainEtl is selected. Native and non-retained captures still fail, and the documented contract says the parent must already exist. Preflight the configured output directory consistently before capture, rather than creating it as a side effect of ETL transfer.
src/core/learning_mode_core/src/paired_output.rs:56
Every capture creates a unique .pair.lock file, but Drop only unlocks it and never removes it. Because capture output names include a per-run ID, these lock files are never reused and accumulate indefinitely in the configured output directory (or the system temp directory for managed outputs). Use an ephemeral/delete-on-close lock strategy, or add race-safe cleanup after the transaction.

impl Drop for OutputPairLock {
    fn drop(&mut self) {
        let _ = std::fs::File::unlock(&self.file);
    }

.github/copilot-instructions.md:216

This architecture entry still says the output layer writes a single JSON document, but this PR makes learning_mode_core own the DataLoopDocument and paired-output transaction as well. Leaving the repository guidance singular will mislead future changes about the required canonical/Data Loop pair; update this entry to describe both artifacts and their paired emission contract.

- `learning_mode_core` is the **cross-platform learning-mode / captureDenials model + output emitter**: `DeniedResource` (+ `ResourceType`/`AccessType`), `DenialSummary`, the `DenialAnalyzer` decode trait, and `emit` — which writes the on-disk denials deliverable as a **single JSON document** `{ "denials": [...], "summary": {...} }` (`write_document` / `DenialsDocument`) and defines the serializable `DenialsOutputPointer`. It carries no OS-specific code (must not depend on any `backends/*` crate); the Windows ETL decoder implementing `DenialAnalyzer` lives in `backends/learning_mode/windows`. When `processContainer.captureDenials` is set, native PSEC/V2 seals and decodes a managed ETL locally, while guarded WPR relogs its host-wide source into a process-scoped ETL before analysis; both routes write the same canonical JSON through shared output plumbing and return neutral `wxc_common` metadata. Explicit `retainEtl` preserves the native sealed trace or the guarded process-scoped relogged trace after a terminal wait; abandonment discards it. `wxc-exec` serializes the metadata as the one-line stderr pointer at the CLI boundary; Rust/C#/FFI callers receive it programmatically. Each denial's `resource` field holds the file path or the AppContainer capability name; capability denials resolve their capability SID to a friendly name via `backends/learning_mode/windows`'s `capability_names` (well-known `S-1-15-3-…` SID → policy name; custom hashed SIDs fall back to the SID string).

Resolve overlapping Learning Mode audit capture changes while preserving Data Loop denial diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab

Darren Hoehna (dhoehna)

Canonical

Does copilot really need to use this every where?

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

docs/learning-mode/capabilities.md:250

This file-denial example does not match the emitted schema. For a decoded access-check containing ObjectName, record_raw_denial_outcome retains that property (redacted) and deliberately does not synthesize resource; consumers following this example will look for a field that is absent from normal file signatures. Use ObjectName here instead.

        "properties": [
          ["PackageSid", "S-1-15-3-1"],
          ["resource", "<REDACTED>"]
        ]

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/backends/learning_mode/windows/src/extractors.rs:152

Event 28 is being classified by the presence of both Category and Detail, not by its TDH schema. A UI-shaped LearningModeViolation event with a malformed/missing Detail therefore falls through to the capability extractor and is logged as capability/notActionable (or unresolvedCapability) instead of a UI payload error. This also contradicts the PR's stated rule that malformed event-28 diagnostics use the schema name to choose UI versus capability. Preserve the schema event name in DecodedEventParts and use it for event-28 routing; property presence can then remain validation within the chosen extractor.

        CAPABILITY_DENIAL_EVENT_ID if is_ui_violation_schema(parts) => {
            build_denial_from_learning_mode(parts, pid, filetime, provider)
        }
        CAPABILITY_DENIAL_EVENT_ID => build_denial_from_capability(parts, pid, filetime, provider),

src/backends/learning_mode/windows/src/extractors.rs:649

Adding SymbolicLink as ResourceType::Other sends it through Accumulator::add_raw_denial's existing non-file normalization (etl_decode.rs:254-256). For the covered object name \??\C:, path_norm::to_user_visible strips the object-manager prefix and the canonical output becomes C:, even though the documentation promises the raw identifier for other resources. This loses namespace information and can merge distinct identifiers. Keep object-manager names raw (or restrict user-visible path normalization to ResourceType::File) before emitting and deduplicating these new denial types.

    let resource_type = match object_type_str {
        "File" => ResourceType::File,
        "Key" => ResourceType::Other,
        "Section" | "SymbolicLink" | "Timer" => ResourceType::Other,
        // A present-but-empty object type is a brokered-capability check.
        "" => ResourceType::Capability,
        _ => return Err(VerboseLoggingExclusionReason::UnsupportedObjectType),

Richie Gomez (richiemsft) added 3 commits August 17, 2026 09:23

Document unified audit architecture

3816503

Update repository guidance for captureDenials-backed audit routing and guarded retainEtl parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Address audit capture review findings

5e1090c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Richie Gomez (richiemsft) force-pushed the unified-audit-capture branch from a300213 to 5e1090c Compare August 17, 2026 16:29

Richie Gomez (richiemsft) added 6 commits August 17, 2026 10:34

Filter guarded WPR traces by process

20a822f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Document signed PLM developer loop

ca6d8d5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Document local PLM signing workflow

565534c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Link Windows SDK signing tools

b9b15a7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Address retained trace review findings

1dd75d1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Keep interactive traces elevated

c6a7353

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4eff3ee2-323c-494d-99b9-f7b54e495216

Richie Gomez (richiemsft) changed the base branch from unified-audit-capture to main August 17, 2026 19:44

Copilot AI balanced review requested due to automatic review settings August 17, 2026 19:54

Richie Gomez (richiemsft) force-pushed the data-loop-decoder branch from dbf31fe to abf1219 Compare August 17, 2026 19:54

microsoft-github-policy-service Bot added the Copilot-Instructions PR modifies Copilot instruction files (.github/copilot-instructions.md or .github/instructions/) label Aug 17, 2026

Copilot started reviewing on behalf of Richie Gomez (richiemsft) August 17, 2026 19:57 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Comment thread src/backends/learning_mode/windows/src/etl_filter.rs Outdated

Comment thread src/host/plm/src/analysis.rs

Comment thread src/backends/learning_mode/windows/src/etl_decode.rs Outdated

Richie Gomez (richiemsft) added 8 commits August 17, 2026 17:23

Copilot AI review requested due to automatic review settings August 18, 2026 02:45

Copilot started reviewing on behalf of Richie Gomez (richiemsft) August 18, 2026 02:46 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Merge main into data-loop-decoder

7f5bf97

Resolve overlapping Learning Mode audit capture changes while preserving Data Loop denial diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9ac49b1-fdfe-4cca-9412-9faa99cef2ab